Methods, systems and apparatus to improve multi-demographic modeling efficiency

ABSTRACT

Methods, apparatus, systems and articles of manufacture are disclosed to improve multi-demographic modeling efficiency. An example apparatus includes a feature set aggregator to segregate training data based on feature sets of interest, and to identify households that participate in at least one of the feature sets of interest, a class enumerator to reduce multi-demographic model iterations by enumerating demographic combinations for the identified households, the enumerated demographic combinations including a single identifier to represent a combination of two or more demographic categories, and a modeling engine to generate training coefficients associated with respective ones of the enumerated demographic combinations.

FIELD OF THE DISCLOSURE

This disclosure relates generally to demographic modeling, and, moreparticularly, to methods, systems and apparatus to improvemulti-demographic modeling efficiency.

BACKGROUND

In recent years, marketing analysts have applied modeling techniques todetermine household demographic compositions. Such modeling techniquesoperate on one or more types of available data sources, such as surveydata and panelist data, the latter of which includes a degree ofmeasurement accuracy that can be used for predictions and/or projectionsthat meet expectations of statistical significance.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic illustration of a demographic modeling systemconstructed in accordance with the teachings of this disclosure toimprove multi-demographic modeling efficiency.

FIG. 2 is an example training data set analyzed by the exampledemographic modeling system of FIG. 1 to estimate household demographicinformation with binary demographic indicators.

FIG. 3 is an example training data set analyzed by the exampledemographic modeling system of FIG. 1 to estimate household demographicinformation with household member count demographic indicators.

FIG. 4 is an example training data set analyzed by the exampledemographic modeling system of FIG. 1 to estimate household demographicinformation in connection with third party models.

FIGS. 5-8 are flowcharts representative of example machine readableinstructions that may be executed to implement the example demographicmodeling system and/or the example demographic modeling manager of FIG.1 to improve multi-demographic modeling efficiency.

FIG. 9 is a block diagram of an example processor platform structured toexecute the example machine readable instructions of FIGS. 5-8 toimplement the example demographic modeling system of FIG. 1.

DETAILED DESCRIPTION

Marketing analysts may estimate household demographic compositions basedon viewing data. For example, by applying retrieved and/or otherwisereceiving viewing data (e.g., from panelist households), models may beapplied to determine the presence or absence of a particular demographicof interest within households of interest. While such modeling effortsreveal valuable demographic insight for the demographic of interest fora particular household, the analyst must then re-run another model forother demographics of interest in a demographic-by-demographic manner.In some examples, marketing analysists seek household demographicinformation for many different demographic categories of interest thatare typically found in households. For instance, some households includetwo parents and one or more children, which requires modeling analysisfor (a) the female (mother) of the household for any number of agecategories (e.g., 19-23, 24-27, 28-31, etc.), the male (father) of thehousehold for any number of (similar) age categories, and (c) the one ormore children for any number of age categories (e.g., 0-6 months, 7months-1 year, 2-5, 6-9, etc.). For each household member, a separatemodel is typically executed to determine demographic details for eachcandidate age category of interest. Accordingly, for even a relativelysmall number of household members, multiple modeling iterations arerequired to obtain demographic estimations in view of the viewing data.

Additionally, after any number of modeling iterations occur for eachcandidate member of a household to determine individual demographics ofthat household, the analyst must also perform additional modelingefforts to identify potential correlation effects. As such, modelingprocessing requirements dramatically increase as a number of householdmembers increases.

Methods, systems, apparatus and articles of manufacture disclosed hereinimprove multi-demographic modeling efficiency. Unlike traditionalapproaches to estimate demographics with viewer data with modeliterations for (a) each demographic of interest and (b) combining themodel iterations with collinearity modeling iteration(s) and/orcorrection efforts, examples disclosed herein apply a single truth modelto the viewer data to generate truth coefficients for classes ofdemographic combinations. Examples disclosed herein model the classesone time to allow test data to be applied to the model to determinecorresponding demographic probabilities for demographics of interestwithout requiring individualized models to be calculated and/orotherwise applied for each demographic of interest. As such, examplesdisclosed herein facilitate technical benefits including, but notlimited to, reducing a number of computational models applied duringmarketing research, and reducing an amount of computational resourceutilization during marketing research efforts. By reducing anapplication of numerous models and/or modeling iterations on ademographic-by-demographic manner, examples disclosed herein improvecomputational efficiency (e.g., reduce or minimize needed computationalresources) as well as improve marketing analyst efficiency (e.g., reduceor minimize an amount of time preparing a marketing model) whenperforming marketing research.

FIG. 1 is a schematic illustration of a demographic modeling system 100.In the illustrated example of FIG. 1, the demographic modeling system100 includes a demographic modeling manager 102 communicativelyconnected to one or more network(s) 104 that is/are communicativelyconnected to a truth data store 106 and a test data store 108. Theexample truth data store 106 includes panelist data of viewing behaviorin which a detailed household composition is known. In some examples,household viewing behaviors stored in the example truth data store 106include tuning behavior data (e.g., channel tuned, channel change data,duration (e.g., in minutes) of channel viewing, etc.) associated withset-top-boxes (STBs). Additionally, the example tuning behavior data isassociated with household demographics (e.g., number of householdmembers, household member age information, household member genderinformation, household member income information, etc.). In someexamples, the tuning behavior is derived from panelist households with aPeopleMeter® (PM) metering device installed by an audience measuremententity (e.g., The Nielsen Company (US), LLC, Experian, etc.) thatmonitors media exposure data (e.g., viewing data, listening data, gamingactivity, etc.) of an enlisted audience. Audience measurement typicallyinvolves determining the identity of the media being displayed and/orotherwise consumed on a media presentation device (e.g., a television),determining demographic information of an audience, and/or determiningwhich members of a household are to be associated with exposure toparticular media. Some audience measurement systems physically connectto the media presentation device, such as the television, to identifywhich channel is being tuned or which program is tuned by capturing achannel number, audio signatures and/or codes identifying (directly orindirectly) the programming being displayed by the media presentationdevice. Additionally, some audience measurement systems prompt and/oraccept audience member input to reveal which household member iscurrently exposed to the media presented by the media presentationdevice.

The example test data store 108 includes audience viewing behaviorinformation from one or more other sources, some of which may not bemanaged to a statistical rigor expected by panelist data sources. Insome examples, the audience viewing behavior information stored in thetest data store 108 is devoid of associated demographic information, butdoes include tuning behavior information. In still other examples, theaudience viewing behavior information stored in the test data store 108includes third party models of demographic viewing behavior. Some suchthird party models may be developed by audience measurement entitiesand/or personnel unassociated with the market analyst using the exampledemographic modeling system 100 of FIG. 1. However, some such thirdparty models may have particular strengths and/or capabilities foraccurately predicting, projecting and/or otherwise estimating viewingbehaviors for particular demographics. As described in further detailbelow, example third party models stored in the example test data store108 may be applied to examples disclosed herein to derive modelcorrections of one or more models generated by the example demographicmodeling manager 102.

The example demographic modeling manager 102 of FIG. 1 also includes anexample data retriever 110, an example feature set segregator 112, anexample household data associator 114, an example class enumerator 116,an example Cartesian combiner 118, an example modeling engine 120, anexample linear combiner 122 and an example condition engine 124. Inoperation, the example data retriever 110 retrieves training data fromthe example truth data store 106. The training data may be associatedwith viewership data and associated demographic information from ageography of interest. In some examples, the marketing analyst isinterested in understanding demographic information associated with oneor more particular features of the viewing audience, such as one or moreparticular programs that have been tuned by and/or otherwise consumed byaudience members. Example features include, but are not limited toparticular programs of interest (e.g., broadcast news programs,situational comedies), particular movies of interest, particular songs,etc. Depending on the features of interest selected by the marketinganalyst, the example feature set segregator 112 segregates the retrievedtraining data, and identifies households that include such activity, asshown in the illustrated example of FIG. 2.

FIG. 2 illustrates example training data 200 retrieved by the exampledata retriever 110 (e.g., from the example truth data store 106) andsegregated by five (5) example features of interest 202. The examplefive features of interest 202 may be indicative of five television showsfor which tuning data occurred for twenty-five (25) example households204. For each intersecting household row 204 and feature column 202, anexample value of quarter hours of viewing is shown. For instance, in theillustrated example of FIG. 2, a first household 206 viewed ten (10)quarter hours of feature 1, four (4) quarter hours of feature 2, eight(8) quarter hours of feature 3, five (5) quarter hours of feature 4, andfive (5) quarter hours of feature 5.

The example household data associator 114 associates availabledemographic data 208 with the respective households 204. In theillustrated example of FIG. 2, two demographics are shown as “D1” 210and “D2” 212. While the illustrated example of FIG. 2 includes 25households, five features and two demographics of interest, suchexamples are included for illustration and not limitation. Exampledisclosed herein may include any number of households, features and/ordemographic of interest to improve multi-demographic modelingefficiency. The example demographic data 208 includes values of one (1)or zero (0) to indicate whether the corresponding household included aparticular demographic of interest when the feature viewing behavioroccurred. Demographic indicators of one (1) reflect the presence of thatparticular demographic, while demographic indicators of zero (0)indicate no viewing behavior of that particular demographic. Forexample, the first household 206 includes a value of 1 for demographicD1 and D2 to indicate that household members of these demographicsparticipated in the corresponding viewing behaviors, and a secondhousehold 213 includes a value of 0 for demographic D1 to indicate thatno household member fitting the qualifications for D1 participated inthe corresponding viewing behaviors, while demographic D2 didparticipate in those viewing behaviors.

As described above, traditional approaches to developing a trainingmodel included the application of training data on each demographic ofinterest individually to generate model coefficients that might be usedfor projection purposes. When one demographic of interest was modeled togenerate corresponding coefficient values, the marketing analystproceeded to identify the next demographic of interest, apply acorresponding model, and generate corresponding coefficients for thatother demographic of interest. After all demographics of interest weremodeled and corresponding coefficients were generated, the analyst againapplied such training data and coefficients to another model to considerpotential effects of collinearity. Accordingly, such traditionalapproaches included multiple iterations of individualized modelingefforts that each consume processing resources, and analystmicro-management.

Unlike traditional approaches to estimate household demographics todevelop a model for predictions and/or projections, examples disclosedherein enumerate unique combinations of participating householddemographics in connection with the training data 200. The example classenumerator 116 enumerates each unique demographic combination to a classrepresentation. In the illustrated example of FIG. 2, because there aretwo separate demographics of interest associated with the training data200, there are three (3) possible unique combinations. In particular,the example class enumerator 116 creates a first class (Class 1) toidentify circumstances in which D1 does not participate in the viewingbehavior, but D2 does participate in the viewing behavior. Additionally,the example class enumerator 116 creates a second class (Class 2) toidentify circumstances in which D1 participates in the viewing behavior,but D2 does not participate in the viewing behavior. Finally, theexample class enumerator 116 creates a third class (Class 3) to identifycircumstances in which both D1 and D2 participate in the viewingbehavior. Note that examples disclosed herein do not considercircumstances in which neither D1 nor D2 participate in the viewingbehavior, as such circumstances do not contribute to the modelingeffort.

The example Cartesian combiner 118 assigns Cartesian products ofcandidate combinations 214, in which coordinates are in a format (D1,D2)having a binary indicator of one (1) or zero (0) to indicate thepresence or absence, respectively, of household members associated withdemographic D1 and/or D2. In the illustrated example of FIG. 2, theCartesian combiner 118 assigns (0,1) as Class 1 to reflect circumstanceswhere D1 does not participate in the viewing behavior, but D2 doesparticipate in the viewing behavior. Additionally, the Cartesiancombiner 118 assigns (1,0) as Class 2 to reflect circumstances where D1does participate in the viewing behavior, but D2 does not participate inthe viewing behavior. Finally, the Cartesian combiner 118 assigns (1,1)as Class 3 to reflect circumstances where both D1 and D2 participate inthe viewing behavior.

The example modeling engine 120 performs a logistic regression of theenumerated class representations to generate probability splits for eachclass representation. As used herein, application of the logisticregression includes a multinomial regression, which is sometimesreferred to herein as a multinomial logistic regression. As shown in theillustrated example of FIG. 2, for each intersecting Cartesian productof candidate combinations 214 and household 204, the example modelingengine 120 generates probability values (e.g., Class 1 probabilityvalues, Class 2 probability values and Class 3 probability values). Theexample first household 206 includes a zero percent probability (seeitem 216) that the Class 1 combination (that is, no participation by ahousehold member matching demographic D1 and participation by ahousehold member matching demographic D2) occurs, a 29% probability (seeitem 218) that the Class 2 combination occurs, and a 71% probability(see item 220) that the Class 3 combination occurs. When the examplemodeling engine 120 applies the logistic regression to the exampletraining data 200 in connection with the enumerated classes 214,corresponding coefficients for each enumerated class of interest result.In the illustrated example of FIG. 2, the modeling engine 120 calculatesa Class 1 model coefficient “A” (see item 222), a Class 2 modelcoefficient “B” (see item 224), and a Class 3 model coefficient “C” (seeitem 226). Such corresponding coefficients are later used in connectionwith test data and/or test sets for the purpose of demographicprojections with data for which the underlying demographic data isunknown or not otherwise verified as true, as described in furtherdetail below.

To estimate and/or otherwise predict a probability of a demographic ofinterest participating in a household of interest, the example linearcombiner 122 applies linear combinations of the example correspondingclass probabilities (e.g., for the example first household 206 the Class1 probability of 0% (see item 216), the Class 2 probability of 29% (seeitem 218), and the Class 3 probability of 71% (see item 220) are used).To illustrate, the example linear combiner 122 selects a household ofinterest (e.g., the first household 206) and a demographic of interestfor which a predicted probability value is desired (e.g., demographicD1). Based on the selected demographic of interest, the example linearcombiner 122 selects class probability values for only those classes inwhich demographic D1 has participated. Recalling that the example classCartesian coordinates are in a binary representation of (D1, D2), thenthe example linear combiner 122 only selects class probability valuesthat include the demographic of interest (e.g., D1). As such, theexample linear combiner 122 selects values associated with Class 2 (29%)and Class 3 (71%) to perform a summation (100%) for the probability thata household member fitting demographic D1 participated in thecorresponding viewing behaviors.

In the event that one or more additional demographics of interest forthe household of interest are to be evaluated, the example linearcombiner 122 selects another demographic of interest (e.g., demographicD2). In this case, only Class 1 (0%) and Class 3 (71%) meet the criteriafor activity associated with demographic D2. The example linear combiner122 performs a summation of these available values to calculate a 71%probability that a household member fitting demographic D2 participatedin the corresponding viewing behaviors. When all demographics for thehousehold of interest have been evaluated, the example linear combiner122 determines if one or more households of interest are to be similarlyevaluated, and the aforementioned calculations repeat as describedabove.

As described above, the model is now trained and ready to be applied toone or more test sets of data to determine predicted probabilities whendemographic information associated with viewing behavior is unknown. Insome examples, the test sets of data are not associated with rigorouscontrol and management typically associated with panelist data. In someexamples, the test sets of data are associated with third party models,the consistency and/or accuracy of which is unknown. Examples disclosedherein evaluate such third party models in view of the training data to,in part, determine whether the third party models are consistent withknown truth data and/or determine whether such third party truth modelsshould be ignored, accepted or accepted in part.

While the illustrated example of FIG. 2 included demographic data 208 inwhich demographic values are a binary indicator (e.g., 0 or 1)indicative of the absence or presence of a demographic of interest,examples disclosed herein are not limited thereto. In some examples,included demographic data may include values indicative of a number ofhousehold members of a particular demographic group that participated inthe viewing behavior.

FIG. 3 illustrates example training data 300 retrieved by the exampledata retriever 110 (e.g., from the example truth data store 106) that,similar to the illustrated example of FIG. 2, is segregated by five (5)example features of interest 302. Similar to the illustrated example ofFIG. 2, the illustrated example of FIG. 3 includes demographic data 308with two separate demographic types of interest; “D1” 310 and “D2” 312.However, unlike the binary representation of demographics shown in theillustrated example of FIG. 2, the values associated with the exampledemographic data 308 of FIG. 3 are indicative of a quantity of memberswithin the respective households that meet the demographic criteria ofinterest. For example, in an example first household 306, D1 has a valueof “1” to indicate one household member fitting the demographic criteriaof D1, and D2 has a value of “2” to indicate two household membersfitting the demographic criteria of D2.

In the illustrated example of FIG. 3, uniquely occurring combinations ofdemographic D1 and demographic D2 can include between zero and threehousehold members. As such, there are a total of sixteen possiblecombinations of household members for D1 and D2 (two demographic values,each with four possible values→2{circumflex over ( )}4=16). However,because examples disclosed herein do not consider the combination wherezero household members were present for D1 and D2, a total of fifteen(15) combinations exist with the illustrated example of FIG. 3. Anexample class column 350 enumerates each uniquely occurring combinationof D1 and D2 as a separate class, beginning with a designating letter“A” (Class A) to represent circumstances in which household combinationshave one member of type D1 and two members of type D2 (e.g., (1,2)).While the unique combination of Class A only occurs once in theillustrated example of FIG. 3 for the example first household 306, anexample Class E occurs five (5) times in household 5, household 10,household 13, household 20 and household 23. In particular, Class Ereflects circumstances in which there are three household members thatsatisfy the demographic criteria of D1 and D2 (e.g., (3,3)). Afterconsidering all unique combinations of D1 and D2 in the illustratedexample of FIG. 3, there are thirteen unique combinations (labeled asClass A through Class M). However, considering that there is a potentialfor fifteen unique combinations of D1 and D2 (in view of zero to threemembers possible in each demographic of interest), the example data 300did not include the missing combinations of (2,1) (e.g., two householdmembers associated with D1 and one household member associated with D2)and (3,0) (e.g., three household members associated with D1, and nohousehold members associated with D2).

Despite the aforementioned differences regarding binary demographicrepresentations in the illustrated example of FIG. 2, and numeric countsof household members for each demographic of interest in the illustratedexample of FIG. 3, the above identified examples to develop a trainingmodel may be accomplished in substantially the same manner. With theexample model coefficients, such as the example Class 1 modelcoefficient A (222), the example Class 2 model coefficient B (224), andthe example Class 3 model coefficient C (226) of FIG. 2, one or moretest sets may be applied thereto.

In operation, the example data retriever 110 retrieves test set data,such as observed behavior data for which corresponding demographicinformation is unknown. In the illustrated example of FIG. 2, the dataretriever 110 retrieved a first test set 270 (e.g., from the exampletest data store 108) of quarter hours of viewing time of the five (5)features of interest 202. The example modeling engine 120 applies themodel, including the example three model coefficients A (222), B (224)and C (226) to derive probability values for each class of interest, asshown in the example test set results matrix 272. The example test setresults matrix 272 includes the three classes described above, whichincludes a Class 1 cell 274, a Class 2 cell 276 and a Class 3 cell 278.While the illustrated example test set results matrix 272 also includesa placeholder for D1=0 and D2=0, such circumstances do not contribute tothis analysis because it does not reveal behavior information.

The result of the example modeling engine 120 performing the multinomialregression on the example first test set 270 in connection with theexample three model coefficients A (222) (associated with Class 1), B(224) (associated with Class 2) and C (226) (associated with Class 3)are probability values of zero, 0.913 and 0.9087, respectively. Asdescribed above, these probability values reflect class probabilitiesthat include component influences from the demographics of interest (D1and D2). The example class enumerator 116 reverts the classes back toprobability values for each demographic of interest by summing theprobability values of the test results matrix 272 according to eachdemographic. In other words, because the classes include two or moreindividual demographic components together (e.g., as a probabilityvalue), the class enumerator 116 reverts those classes intoindividualized demographic components, each having an associatedprobability value. For example, cells of the example test results matrix272 that are associated with D1 behaviors include the Class 2 cell (276)and the Class 3 cell (278). The sum of resulting probability values ofthese cells adds up to 1.00, which reveals that there is a 100%probability that the test set data 270 behaviors included a householdmember belonging to demographic D1. Similarly, cells of the test resultsmatrix 272 that are associated with D2 behaviors include the Class 1cell (274) and the Class 3 cell (278). The sum of resulting probabilityvalues of these cells adds up to 0.9087, which reveals that there is a91% probability that the test set data 270 behaviors included ahousehold member belonging to demographic D2.

As described above, examples disclosed herein also generate models inwhich a training set includes third party models. For example, Experian®collects information on people, businesses, motor vehicles, insuranceand lifestyle data (e.g., via online surveys, offline surveys, socialmedia monitoring, etc.). In some examples, the third party models (e.g.,from Experian®) exhibit particular strengths to reveal demographicinformation associated with particular demographics of interest.However, in some examples, the third party models may not align well orotherwise satisfy statistical requirements to allow predictions,projections and/or estimations.

FIG. 4 illustrates example training data 400 retrieved by the exampledata retriever 110 that, similar to the illustrated examples of FIGS. 2and 3, is segregated by five (5) example features of interest 402.Similar to the illustrated examples of FIGS. 2 and 3, the illustratedexample of FIG. 4 includes demographic data 408A with two separatedemographic types of interest associated with the training data 400;“D1” 410A and “D2” 412A. However, the illustrated example of FIG. 4 alsoincludes a third party model, which includes demographic data 408B withrepresentations of the same two separate demographic types of interest;“D1” 410B and “D2” 412B. In some examples, third party model data may bestored in the example test data store 108.

As described above, the example class enumerator 116 enumerates eachunique demographic combination to a class representation, and theexample Cartesian combiner 118 assigns Cartesian products of candidatecombinations. In the illustrated example of FIG. 4, nine (9) separateCartesian products of candidate combinations 414 are shown havingcoordinates in a format (D1,D2) with a binary indicator of one (1) orzero (0) to indicate the presence or absence, respectively, of householdmembers associated with demographic D1 and/or D2. Additionally, becausethe illustrated example of FIG. 4 considers a comparison to a thirdparty model, Cartesian coordinate intersections of interest between thetruth model and the third party model are analyzed. For example, a firstintersecting column of interest 450 includes an intersection of thethird party model in which D1 does not participate and D2 doesparticipate in the viewing behavior (0,1) with the truth model in whichD1 does not participate and D2 does participate with the viewingbehavior (0,1). Taken together, the first intersecting column ofinterest 450 is shown as “(0,1) (0,1).” Additional columns of interestare shown in the illustrated example of FIG. 4 for the remainingcandidate combinations of interest between the truth model and the thirdparty model, and there are nine (9) columns in total becausecircumstances where there is no viewing behavior for D1 and D2 are notconsidered in this analysis (e.g., (0,0) (0,1), (0,0) (1,0), (1,1)(0,0), etc.).

As described above, the example modeling engine 120 performs amultinomial logistic regression of the enumerated class representationsin connection with the training data 400 and the third party model togenerate probability splits for each class representation. In theillustrated example of FIG. 4, the modeling engine 120 calculates a 53%probability that the combination (0,1) (0,1) occurs in connection withthe associated training data 400. In other words, there is a 53%probability that (a) the third party model predicts that zero householdmembers associated with D1 and one household member associated with D2participate in the viewing behavior and (b) the truth model predicts thesame.

In addition to calculating probability values for the classintersections, the example modeling engine 120 also calculatescoefficients for each class of interest to be used for estimations inthe event one or more test data sets are obtained. However, examplesdisclosed herein also reveal valuable information related to the thirdparty model regarding its ability to accurately predict particular typesof demographic behavior. As described above, the example linear combiner122 applies linear combinations of the calculated class probabilities toestimate a likelihood that a particular demographic is present. In theillustrated example of FIG. 4, results of demographic likelihoods forthe third party model and the truth model 452 are shown. These exampleresults 452 illustrate particular strengths or weaknesses of the thirdparty model for predicting demographic behaviors for particulardemographics of interest. For example, an example second household 413illustrates that the modeling engine 120 and linear combiner 122 havecalculated the third party model to predict a 69% probability for D1 anda 61% probability for D2. Additionally, the example modeling engine 120and the linear combiner 122 have calculated the truth model to predict a71% probability for D1 and a 63% probability for D2. The example linearcombiner 122 compares these predictions to determine difference valuesand, in this case, the two models differ by 2% for D1 and 2% for D2. Insome examples, the linear combiner 122 compares these differences to athreshold value that, when satisfied (e.g., exceeded), generates a flagto warn against and/or otherwise prevent the third party model frombeing relied upon for model correction or use. In other words, theexample linear combiner 122 facilitates an ability to generate a trustmetric for the third party model.

To illustrate, consider an example fourth household 454 that illustratesthe third party model predicts a 57% probability for D1, while the truthmodel predicts a 25% probability for D1. In this example, the thirdparty model overpredicted demographic D1 by 32%. In the event adifference threshold were established at, for example, 10%, then theexample linear combiner 122 would flag the third party model asunreliable for these circumstances.

However, in some examples additional information associated with thethird party model is available to reveal that a particular demographiccombination was empirically observed. In the event that the third partymodel also includes observation data (e.g., empirical observation data)for a particular demographic combination (e.g., D1=1, D2=0), then theexample modeling engine 120 adjusts the truth model to align closer tothe third party model (e.g., weighting).

While an example manner of implementing the demographic modeling system100 of FIG. 1 is illustrated in FIG. 1, one or more of the elements,processes and/or devices illustrated in FIG. 1 may be combined, divided,re-arranged, omitted, eliminated and/or implemented in any other way.Further, the example truth data store 104, the example test data store106, the example data retriever 110, the example feature set segregator112, the example household data associator 114, the example classenumerator 116, the example Cartesian combiner 118, the example modelingengine 120, the example linear combiner 122, the example conditionengine 124 and/or, more generally, the example demographic modelingmanager 102 of FIG. 1 may be implemented by hardware, software, firmwareand/or any combination of hardware, software and/or firmware. Thus, forexample, any of the example truth data store 104, the example test datastore 106, the example data retriever 110, the example feature setsegregator 112, the example household data associator 114, the exampleclass enumerator 116, the example Cartesian combiner 118, the examplemodeling engine 120, the example linear combiner 122, the examplecondition engine 124 and/or, more generally, the example demographicmodeling manager 102 of FIG. 1 could be implemented by one or moreanalog or digital circuit(s), logic circuits, programmable processor(s),application specific integrated circuit(s) (ASIC(s)), programmable logicdevice(s) (PLD(s)) and/or field programmable logic device(s) (FPLD(s)).When reading any of the apparatus or system claims of this patent tocover a purely software and/or firmware implementation, at least one ofthe example truth data store 104, the example test data store 106, theexample data retriever 110, the example feature set segregator 112, theexample household data associator 114, the example class enumerator 116,the example Cartesian combiner 118, the example modeling engine 120, theexample linear combiner 122, the example condition engine 124 and/or,more generally, the example demographic modeling manager 102 of FIG. 1is/are hereby expressly defined to include a tangible computer readablestorage device or storage disk such as a memory, a digital versatiledisk (DVD), a compact disk (CD), a Blu-ray disk, etc. storing thesoftware and/or firmware. Further still, the example demographicmodeling system 100 of FIG. 1 may include one or more elements,processes and/or devices in addition to, or instead of, thoseillustrated in FIG. 1, and/or may include more than one of any or all ofthe illustrated elements, processes and devices.

Flowcharts representative of example machine readable instructions forimplementing the demographic modeling system 100 of FIG. 1 are shown inFIGS. 5-8. In these examples, the machine readable instructions comprisea program for execution by a processor such as the processor 912 shownin the example processor platform 900 discussed below in connection withFIG. 9. The program may be embodied in software stored on a tangiblecomputer readable storage medium such as a CD-ROM, a floppy disk, a harddrive, a digital versatile disk (DVD), a Blu-ray disk, or a memoryassociated with the processor 912, but the entire program and/or partsthereof could alternatively be executed by a device other than theprocessor 912 and/or embodied in firmware or dedicated hardware.Further, although the example program is described with reference to theflowcharts illustrated in FIGS. 5-8, many other methods of implementingthe example demographic modeling system 100 may alternatively be used.For example, the order of execution of the blocks may be changed, and/orsome of the blocks described may be changed, eliminated, or combined.

As mentioned above, the example processes of FIGS. 5-8 may beimplemented using coded instructions (e.g., computer and/or machinereadable instructions) stored on a tangible computer readable storagemedium such as a hard disk drive, a flash memory, a read-only memory(ROM), a compact disk (CD), a digital versatile disk (DVD), a cache, arandom-access memory (RAM) and/or any other storage device or storagedisk in which information is stored for any duration (e.g., for extendedtime periods, permanently, for brief instances, for temporarilybuffering, and/or for caching of the information). As used herein, theterm tangible computer readable storage medium is expressly defined toinclude any type of computer readable storage device and/or storage diskand to exclude propagating signals and to exclude transmission media. Asused herein, “tangible computer readable storage medium” and “tangiblemachine readable storage medium” are used interchangeably. Additionallyor alternatively, the example processes of FIGS. 5-8 may be implementedusing coded instructions (e.g., computer and/or machine readableinstructions) stored on a non-transitory computer and/or machinereadable medium such as a hard disk drive, a flash memory, a read-onlymemory, a compact disk, a digital versatile disk, a cache, arandom-access memory and/or any other storage device or storage disk inwhich information is stored for any duration (e.g., for extended timeperiods, permanently, for brief instances, for temporarily buffering,and/or for caching of the information). As used herein, the termnon-transitory computer readable medium is expressly defined to includeany type of computer readable storage device and/or storage disk and toexclude propagating signals and to exclude transmission media. As usedherein, when the phrase “at least” is used as the transition term in apreamble of a claim, it is open-ended in the same manner as the term“comprising” is open ended.

The program 500 of FIG. 5 begins at block 502, where the example dataretriever 110 retrieves and/or otherwise receives training data from theexample truth data store 106. Depending on the number and/or type offeatures of interest to be analyzed, the example feature set segregator112 segregates the retrieved training data based on those feature setsof interest (block 504). In some examples, one or more feature sets areselected to help reveal audience behavior(s) and/or trends related tobroadcast assets, such as particular television shows or particularcommercials, etc. For example, the marketing analyst and/or a client ofthe marketing analyst may be interested in learning about whichdemographic household members tend to consume and/or otherwise view theparticular television programs broadcast on media devices to a greateror lesser degree. For those particular programs (e.g., features), theexample feature set segregator 112 identifies particular households fromthe training set that include viewing activity associated with thefeature sets of interest, and includes viewing data associated therewith(block 506). As such, the viewing data to be included in the analysis isfocused on (a) the feature(s) of interest and (b) the viewing data forwhich those features have occurred. In some examples, the viewing dataincludes a number of quarter-hours of viewing time, as shown in FIGS.2-4.

The segregated viewing data from the example truth data store 106 ispanelist data that also includes verified and/or otherwise truedemographic information. The example household data associator 114associates available demographic data with each household of interest(block 508), which is shown as item 208 in FIG. 2. The example classenumerator 116 enumerates each unique demographic combination to a classrepresentation (block 510). For example, a combination of a householdmember associated with demographic D1 not viewing and a household memberassociated with demographic D2 viewing is enumerated as “Class 1.” Asanother example, a combination of a household member associated withdemographic D1 viewing and another household member associated withdemographic D2 viewing is enumerated as “Class 3.” The enumeratedclasses are assigned, by the example Cartesian combiner 118, Cartesianproduct representations of candidate combinations (block 512). Examplesof such enumerated assignments are shown as item 214 of FIG. 2.

The example modeling engine 120 performs a conditional maximum entropymodeling of the truth data and enumerated combinations by performing alogistic regression (e.g., a multinomial logistic regression) of theenumerated class representations (block 514). In particular, the examplemodeling engine 120 generates probability splits for each classrepresentation for all participating households, and verifies that allhousehold probability values add up to 100%. Additionally, the examplemodeling engine 120 fits the multinomial regression to generate modelcoefficients associated with each unique enumerated class (block 516),such as the example Class 1 model coefficient “A” (222), the exampleClass 2 model coefficient “B” (224), and the example Class 3 modelcoefficient “C” (226) described above in connection with FIG. 2. Asdescribed above and in further detail below, the example modelcoefficients may be used when generating estimates in connection withtest sets of data for which demographic information is unknown.

While the example model class probabilities resulted from theapplication of the multinomial regression, such probabilities are notyet in terms of the demographic of interest, but rather two or moredemographic combinations that may have occurred with the training(truth) data set (e.g., the example training data 200 of FIG. 2, theexample training data 300 of FIG. 3, the example training data 400 ofFIG. 4, etc.). To estimate the likelihood that a particular demographicis present, the example linear combiner 122 applies linear combinationsof the class probabilities that represent a target demographic ofinterest (block 518).

FIG. 6 illustrates additional detail in connection with applying alinear combination of interest of block 518. In the illustrated exampleof FIG. 6, the example linear combiner 122 selects a household ofinterest (block 602) having associated class probability values. Toillustrate, examples from FIG. 2 are described, but similar processesmay be applied to the illustrated examples of FIGS. 3 and/or 4. Assumingfor this example that a predicted probability for demographic D1 isdesired (block 604), the linear combiner 122 selects values from onlythose enumerated classes in which D1 occurs (block 606). For example,Class 1 includes a Cartesian representation (0,1), Class 2 includes aCartesian representation (1,0), and Class 3 includes a Cartesianrepresentation (1,1). For each example Cartesian representation, thefirst value reflects the occurrence or non-occurrence of demographic D1,and the second value reflects the occurrence or non-occurrence ofdemographic D2. For the sake of this example, in which D1 is the targetdemographic of interest, the example linear combiner 122 selects onlyClass 2 and Class 3 because they each include potential behavior dataassociated with D1 (e.g., those classes include a value of “1” in theCartesian representation for D1). In the illustrated example of FIG. 2,the sum of the values for Class 2 (0.29) and Class 3 (0.71) is 1.00,which reflects a 100% probability that a household member associatedwith D1 participated in the activity (e.g., viewing behavior).

Values for those classes of interest are summed (block 608) to yield apredicted probability for the demographic of interest. The examplelinear combiner 122 determines whether an additional demographic ofinterest is available for the selected household (block 610). If so,control returns to block 604 to select an alternate/additionaldemographic of interest. If not, then the linear combiner 122 determineswhether there are additional households left to evaluate (block 612). Ifso, then control returns to block 602, otherwise the example program 518ends and control returns to block 520 of FIG. 5.

As described above, after the training data has been modeled to generatecoefficients associated with each unique enumerated class of demographiccombinations, one or more test sets of data may be applied to the modelto estimate corresponding demographic probabilities associated withthose test set(s). In some examples, a test set may include viewingbehavior for features of interest, such as quantities of viewed quarterhours of particular television shows. In some examples, application ofthe test set occurs after the training data has been modeled inconnection with a third party model, as shown in the illustrated exampleof FIG. 4. In some examples, it is desired to determine whether thethird party model can be trusted, which may include determiningparticular demographics of interest that the third party model estimatesrelatively well as opposed to particular demographics of interest thatthe third party model does not estimate well.

The example demographic modeling manager 102 determines whether toevaluate a third party model (block 520) and, if so, the examplemodeling engine evaluates the third party model (block 522) on ademographic-by-demographic basis to ascertain how well or how poorly itperforms when compared to the truth model (which is based onpanelist-based training data). FIG. 7 illustrates additional detail toevaluate the third party model in connection with block 522. In theillustrated example of FIG. 7, the example linear combiner 122 comparesprobability results between the third party model and the training modelfor demographic of interest (block 702). Depending on whether the thirdparty model also has actual (e.g., empirical) data associated with thedemographic of interest, examples disclosed herein will either use thethird party model results to help correct the training model or not. Theexample data retriever 110 determines whether actual truth data isavailable from the third party model for the demographic of interest(block 704) and, if so, the example modeling engine 120 adjusts thetraining model with a weighted result of the probability result of thethird party model (block 706). In some examples, the weighted result isbased on the probability difference value between the third party modeland the training model.

On the other hand, if there is no trust data available from the thirdparty model associated with the demographic of interest (block 704),then the example linear combiner determines whether the deviation(difference) satisfies (e.g., exceeds) a threshold value (block 708). Insome examples, if the difference in probability values estimated by thethird party model and the training model exceeds the threshold (e.g.,10%), then the third party model validity and/or capability is deemedless reliable (block 710). However, if the difference in probabilityvalues (for the demographic of interest) estimated by the third partymodel and the training model does not exceed the threshold (block 708),then the example modeling engine 120 flags the third party model asreliable (block 712). The linear combiner determines whether additionaldemographics of interest are to be compared (block 714) and, if so,control returns to block 702.

Returning to the illustrated example of FIG. 5, if there is no thirdparty model to be evaluated (block 520), or after such a third partymodel has been evaluated (block 522), the example demographic modelingmanager 102 applies one or more test sets of data to the training model(block 524). FIG. 8 illustrates additional detail of applying test setsin connection with block 524. In the illustrated example of FIG. 8, theexample data retriever 110 retrieves test data (block 802), such as theexample first test set 270 of quarter hours of viewing time shown inFIG. 2. The example modeling engine 120 applies the training model(e.g., the example Class 1 coefficient “A” 222, the example Class 2coefficient “B” 224, and the example Class 3 coefficient “C” 226 of FIG.2) to the test set to derive probability values for each class (block804). As described above, classes include combinations of demographicmembers that can later be reverted to individualized demographicprobabilities. The example class enumerator 116 reverts the estimatedclass probabilities back to individualized demographic probabilities foreach demographic of interest (block 806). Control then returns to theillustrated example of FIG. 5 where the process 500 can repeat, asdesired.

FIG. 9 is a block diagram of an example processor platform 900 capableof executing the instructions of FIGS. 5-8 to implement the demographicmodeling system 100 of FIG. 1. The processor platform 900 can be, forexample, a server, a personal computer, a mobile device (e.g., a cellphone, a smart phone, a tablet such as an iPad™), a personal digitalassistant (PDA), an Internet appliance, a gaming console, a set top box,or any other type of computing device.

The processor platform 900 of the illustrated example includes aprocessor 912. The processor 912 of the illustrated example is hardware.For example, the processor 912 can be implemented by one or moreintegrated circuits, logic circuits, microprocessors or controllers fromany desired family or manufacturer. In the illustrated example of FIG.9, the processor 900 includes one or more example processing cores 915configured via example instructions 932, which include the exampleinstructions of FIGS. 5-8 to implement the example demographic modelingsystem 100 of FIG. 1.

The processor 912 of the illustrated example includes a local memory 913(e.g., a cache). The processor 912 of the illustrated example is incommunication with a main memory including a volatile memory 914 and anon-volatile memory 916 via a bus 918. The volatile memory 914 may beimplemented by Synchronous Dynamic Random Access Memory (SDRAM), DynamicRandom Access Memory (DRAM), RAMBUS Dynamic Random Access Memory (RDRAM)and/or any other type of random access memory device. The non-volatilememory 916 may be implemented by flash memory and/or any other desiredtype of memory device. Access to the main memory 914, 916 is controlledby a memory controller.

The processor platform 900 of the illustrated example also includes aninterface circuit 920. The interface circuit 920 may be implemented byany type of interface standard, such as an Ethernet interface, auniversal serial bus (USB), and/or a PCI express interface.

In the illustrated example, one or more input devices 922 are connectedto the interface circuit 920. The input device(s) 922 permit(s) a userto enter data and commands into the processor 1012. The input device(s)can be implemented by, for example, an audio sensor, a microphone, acamera (still or video), a keyboard, a button, a mouse, a touchscreen, atrack-pad, a trackball, isopoint and/or a voice recognition system.

One or more output devices 924 are also connected to the interfacecircuit 920 of the illustrated example. The output devices 924 can beimplemented, for example, by display devices (e.g., a light emittingdiode (LED), an organic light emitting diode (OLED), a liquid crystaldisplay, a cathode ray tube display (CRT), a touchscreen, a tactileoutput device, a printer and/or speakers). The interface circuit 920 ofthe illustrated example, thus, typically includes a graphics drivercard, a graphics driver chip or a graphics driver processor.

The interface circuit 920 of the illustrated example also includes acommunication device such as a transmitter, a receiver, a transceiver, amodem and/or network interface card to facilitate exchange of data withexternal machines (e.g., computing devices of any kind) via a network926 (e.g., an Ethernet connection, a digital subscriber line (DSL), atelephone line, coaxial cable, a cellular telephone system, etc.).

The processor platform 900 of the illustrated example also includes oneor more mass storage devices 928 for storing software and/or data.Examples of such mass storage devices 928 include floppy disk drives,hard drive disks, compact disk drives, Blu-ray disk drives, RAIDsystems, and digital versatile disk (DVD) drives. In some examples, themass storage device may implement the example truth data store 106and/or the example test data store 108.

The coded instructions 932 of FIGS. 5-8 may be stored in the massstorage device 928, in the volatile memory 914, in the non-volatilememory 916, and/or on a removable tangible computer readable storagemedium such as a CD or DVD.

From the foregoing, it will be appreciated that the above disclosedmethods, apparatus and articles of manufacture improve the efficiencywhen modeling multiple demographics of interest from training data sothat estimates may be generated from one or more test sets of data. Inparticular, traditional approaches to model a first demographic ofinterest include a first modeling iteration in view of only thatparticular demographic of interest. Once that modeling effort has beencompleted, a subsequent modeling effort (e.g., using the same trainingdata) is performed on the alternate demographic of interest. Thisprocess repeats for all demographics of interest that are of interestand, when corresponding coefficients for the demographics of interesthave been calculated (e.g., fitted), further modeling efforts must beapplied to compensate for potential modeling errors caused byco-linearity. However, examples disclosed herein reduce (e.g., minimize)a number of modeling iterations for training data in which two or moredemographics of interest are to be estimated. Additionally, examplesdisclosed herein reduce modeling errors in connection with reliance uponthird party models, in which particular estimation and/or predictioncapabilities are unknown. Despite an analyst belief or “word of mouth”that a particular third party model is helpful for demographic research,overreliance on such models when they fail to produce accurate estimatescauses wasted research efforts and computational waste.

Although certain example methods, apparatus and articles of manufacturehave been disclosed herein, the scope of coverage of this patent is notlimited thereto. On the contrary, this patent covers all methods,apparatus and articles of manufacture fairly falling within the scope ofthe claims of this patent.

1. (canceled) 2.-20. (canceled)
 21. An apparatus to reduce computationalresources for a multi-demographic training model, comprising: memory;and at least one processor to execute computer readable instructions toat least: reduce multi-demographic model iterations by enumeratingdemographic combinations for households that access at least one of aplurality of features represented in training data, the enumerateddemographic combinations including an identifier, the identifier torepresent a combination of two or more demographic categories and toinclude cartesian representations corresponding to occurrences ornon-occurrences of a demographic category of interest, the demographiccategory of interest to be selected from the two or more demographiccategories; and generate probability values for ones of the enumerateddemographic combinations based on training coefficients, the probabilityvalues to represent likelihoods that the ones of the enumerateddemographic combinations accessed the at least one of the plurality offeatures, the training coefficients corresponding to the ones of theenumerated demographic combinations, the ones of the enumerateddemographic combinations associated with (a) a training model and (b) athird-party model.
 22. The apparatus as defined in claim 21, wherein theat least one processor is to execute the computer readable instructionsto identify ones of the households that access the at least one of thefeatures of interest.
 23. The apparatus as defined in claim 22, whereinthe at least one processor is to execute the computer readableinstructions to associate demographic combinations with the ones of thehouseholds.
 24. The apparatus as defined in claim 21, wherein the atleast one processor is to execute the computer readable instructions togenerate the training coefficients based on a multinomial logisticregression with: (a) the training data, and (b) the ones of theenumerated demographic combinations.
 25. The apparatus as defined inclaim 21, wherein the at least one processor is to execute the computerreadable instructions to revert the probability values for respectiveones of the enumerated demographic combinations to second probabilityvalues associated with individual demographic components.
 26. Theapparatus as defined in claim 21, wherein the at least one processor isto execute the computer readable instructions to determine probabilityvalue differences between the training model and the third-party model.27. The apparatus as defined in claim 26, wherein the at least oneprocessor is to execute the computer readable instructions to determinea trust metric for the third-party model based on a comparison of theprobability value differences to a threshold, the trust metric toindicate the third-party model is reliable.
 28. A tangiblecomputer-readable storage medium comprising instructions that, whenexecuted, cause a processor to, at least: reduce multi-demographic modeliterations by enumerating demographic combinations for households thataccess at least one of a plurality of features represented in trainingdata, the enumerated demographic combinations including an identifier,the identifier to represent a combination of two or more demographiccategories and to include cartesian representations corresponding tooccurrences or non-occurrences of a demographic category of interest,the demographic category of interest to be selected from the two or moredemographic categories; and generate probability values for ones of theenumerated demographic combinations based on training coefficients, theprobability values to represent likelihoods that the ones of theenumerated demographic combinations accessed the at least one of theplurality of features, the training coefficients corresponding to theones of the enumerated demographic combinations, the ones of theenumerated demographic combinations associated with (a) a training modeland (b) a third-party model.
 29. The computer-readable medium as definedin claim 28, wherein the instructions are to cause the processor toidentify ones of the households that access the at least one of thefeatures of interest.
 30. The computer-readable medium as defined inclaim 29, wherein the instructions are to cause the processor toassociate demographic combinations with the ones of the households. 31.The computer-readable medium as defined in claim 28, wherein theinstructions are to cause the processor to generate the trainingcoefficients based on a multinomial logistic regression with: (a) thetraining data, and (b) the ones of the enumerated demographiccombinations.
 32. The computer-readable medium as defined in claim 28,wherein the instructions are to cause the processor to revert theprobability values for respective ones of the enumerated demographiccombinations to second probability values associated with individualdemographic components.
 33. The computer-readable medium as defined inclaim 28, wherein the instructions are to cause the processor todetermine probability value differences between the training model andthe third-party model.
 34. The computer-readable medium as defined inclaim 33, wherein the instructions are to cause the processor todetermine a trust metric for the third-party model based on a comparisonof the probability value differences to a threshold, the trust metric toindicate the third-party model is reliable.
 35. An apparatus to reducecomputational resources for a multi-demographic training model,comprising: means for reducing multi-demographic model iterations byenumerating demographic combinations for households that access at leastone of a plurality of features represented in training data, theenumerated demographic combinations including an identifier, theidentifier to represent a combination of two or more demographiccategories and to include cartesian representations corresponding tooccurrences or non-occurrences of a demographic category of interest,the demographic category of interest to be selected from the two or moredemographic categories; and means for modeling to generate probabilityvalues for ones of the enumerated demographic combinations based ontraining coefficients, the probability values to represent likelihoodsthat the ones of the enumerated demographic combinations accessed the atleast one of the plurality of features, the training coefficientscorresponding to the ones of the enumerated demographic combinations,the ones of the enumerated demographic combinations associated with (a)a training model and (b) a third-party model.
 36. The apparatus asdefined in claim 35, further including means for aggregating to identifyones of the households that access the at least one of the features ofinterest.
 37. The apparatus as defined in claim 36, wherein the modelingmeans is to generate the training coefficients based on a multinomiallogistic regression with: (a) the training data, and (b) the ones of theenumerated demographic combinations.
 38. The apparatus as defined inclaim 35, further including means for reverting the probability valuesfor respective ones of the enumerated demographic combinations to secondprobability values associated with individual demographic components.39. The apparatus as defined in claim 38, wherein the reverting means isto determine probability value differences between the training modeland the third-party model.
 40. The apparatus as defined in claim 39,wherein the reverting means is to determine a trust metric for thethird-party model based on a comparison of the probability valuedifferences to a threshold, the trust metric to indicate the third-partymodel is reliable.